Nomura modernized their on-premises data lake delivering unparalleled efficiency with Dremio

The Business:

Nomura is a prominent global financial services firm headquartered in Tokyo, Japan. Specializing in investment banking, securities, wealth and asset management, and retail banking, Nomura serves a diverse client base across the world. The company prides itself on delivering comprehensive financial solutions and strategic insights tailored to meet the evolving needs of its clients. With a strong emphasis on innovation and leveraging its extensive market knowledge and networks, Nomura aims to drive growth and create value while maintaining a commitment to responsible business practices.

Founded on a legacy of financial expertise and a deep-rooted presence in global markets, Nomura is dedicated to fostering economic prosperity and contributing positively to society. Through its operations, Nomura strives to uphold high standards of integrity, transparency, and sustainability, aiming to build enduring relationships with clients and stakeholders alike. As a leader in the financial services industry, Nomura continues to innovate and adapt to the dynamic landscape of global finance, positioning itself as a trusted partner in navigating complex financial challenges and opportunities.

The Challenge:

Nomura faced several significant challenges in optimizing its data infrastructure, particularly centered around its use of Hadoop. Initially, Nomura recognized that its relational databases were inadequate to handle the growing data volumes and complex queries required for market and credit risk analysis. This led them to adopt Dremio five years ago, which provided faster query performance and flexibility compared to traditional databases. However, maintaining stability and meeting stringent SLAs became increasingly challenging due to outdated infrastructure and limited insights into system performance and query behavior.

Key challenges included frequent production issues and serious outages, exacerbated by the difficulty in diagnosing problems and the lack of comprehensive monitoring tools. Nomura's Grafana dashboard, initially rudimentary, evolved to include executor-level metrics crucial for monitoring distributed system performance. Additionally, they struggled with the inefficiencies of Hadoop, such as high operational costs and the complexity of managing a monolithic stack, which hindered their ability to scale and innovate effectively.

The Solution:

Nomura implemented several solutions to address their data infrastructure challenges. They enhanced their Grafana dashboard to provide executor-level metrics for better system insight. They created a structured metadata layer and developed a query replay system to emulate production workloads in non-production environments, improving regression testing. Nomura also transitioned from Hadoop to MinIO, utilizing Kubernetes for scalability, which allowed them to repurpose existing hardware. Additionally, they adopted Apache Iceberg to simplify their data pipeline and improve data management capabilities.

Dremio played a crucial role in Nomura's solution by serving as their data processing engine. Initially, Nomura adopted Dremio to handle their large datasets and demanding query workloads. Over time, they integrated Dremio more deeply by enhancing their monitoring and diagnostic capabilities with an evolved Grafana dashboard. They also developed a metadata layer to better manage queries and implemented a query replay system for improved testing. Dremio's performance and flexibility were pivotal in Nomura's data infrastructure improvements and transition to new technologies like MinIO and Apache Iceberg.